Associative Data Storage and Retrieval in Neural Networks 1.1 Introduction and Overview

Associative storage and retrieval of binary random patterns in various neural net models with one-step threshold-detection retrieval and local learning rules are the subject of this paper. For diierent hetero-association and auto-association memory tasks, speciied by the properties of the pattern sets to be stored and upper bounds on the retrieval errors, we compare the performance of various models of nite as well as asymptoti-cally innnite size. In innnite models, we consider the case of asymptotically sparse patterns, where the mean activity in a pattern vanishes, and study two asymptotic delity requirements: constant error probabilities and vanishing error probabilities. A signal-to-noise ratio analysis is carried out for one retrieval step where the calculations are comparatively straightforward and easy. As performance measures we propose and evaluate information capacities in bits/synapse which also take into account the important property of fault tolerance. For auto-association we compare one-step and xed-point retrieval that is analyzed in the literature by methods of statistical mechanics. With growing experimental insight in the anatomy of the nervous system as well as the rst electrophysiological recordings of nerve cells in the rst half of this century, a new theoretical eld was opened, namely, the modelling of the experimental ndings at one or a few nerve cells, leading to very detailed models of biological neurons 1]. But diierent from most biological phenomena, where the macroscopic function can be understood by revealing the cellular mechanism, the function of the nervous system as a whole turned out to be constituted by the collective behaviour of a very large number of nerve cells and the activity of a large fraction of cells, a ii 1. Associative Data Storage and Retrieval in Neural Networks whole activity pattern, had to be considered instead. The modelling had to drop the biological faithfullness at two points: on the cellular level the models had to be simpliied such that a large number of nerve cells could be described and on the macroscopic level the function had to be reduced to simple activity pattern processing like pattern completion, pattern recognition or pattern classiication allowing a theoretical description and quantiication. McCulloch and Pitts 2] argued that due to the \all or none" character of nervous activity the neurophysiological ndings can be reproduced in models with simple two-state neurons, in particular, in associative memory models which exhibit binary activity patterns. In the ftees and sixtees small feed-forward neural nets have been …